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Abstract 
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I.  Introduction 


Convexity  plays  a  central  role  in  a  wide  variety  of  machine  learning  and  statistical  inference  prob¬ 
lems.  A  standard  paradigm  is  to  distinguish  a  preferred  member  from  a  set  of  candidates  based 
upon  a  convex  impurity  measure  or  loss  function  tailored  to  the  specific  problem  to  be  solved. 
Examples  include  least  squares  regression,  decision  trees,  boosting,  online  learning,  maximum  like¬ 
lihood  for  exponential  models,  logistic  regression,  maximum  entropy,  and  support  vector  machines. 
Such  problems  can  often  be  naturally  cast  as  convex  optimization  problems  involving  a  Bregman 
distance,  which  can  lead  to  new  algorithms,  analytical  tools,  and  insights  derived  from  the  powerful 
methods  of  convex  analysis. 

In  this  paper  we  formulate  and  prove  a  convex  duality  theorem  for  minimizing  a  general  class 
of  Bregman  distances  subject  to  linear  constraints.  The  duality  result  is  then  used  to  derive 
iterative  algorithms  for  solving  the  associated  optimization  problem.  Our  presentation  is  motivated 
by  the  recent  work  of  Collins,  Schapire,  and  Singer  (2001),  who  showed  how  certain  boosting 
algorithms  and  maximum  likelihood  logistic  regression  can  be  unified  within  the  framework  of 
Bregman  distances.  In  particular,  specific  instances  of  the  results  given  here  are  used  by  Collins 
et  al.  (2001)  to  show  the  convergence  of  a  family  iterative  algorithms  for  minimizing  the  exponential 
or  logistic  loss. 

While  invoking  methods  from  convex  analysis  can  unify  and  clarify  the  relationship  between  differ¬ 
ent  methods,  the  higher  level  of  abstraction  often  comes  at  a  price,  since  there  can  be  considerable 
technicalities.  For  example,  in  some  treatments  the  assumptions  on  the  convex  functions  that  can 
be  used  to  define  Bregman  distances  are  very  technical  and  difficult  to  verify.  Here  we  trade  off 
generality  for  relative  simplicity  by  working  with  a  restricted  class  of  Bregman  distances,  which 
however  includes  many  of  the  examples  that  arise  in  machine  learning.  Our  treatment  of  duality 
and  auxiliary  functions  for  Bregman  distances  closely  parallels  the  results  presented  by  Della  Pietra 
et  al.  (1997)  for  the  Kullback-Leibler  divergence.  In  particular,  the  statement  and  proof  of  the 
duality  theorem  given  in  (Della  Pietra  et  al.,  1997)  carries  over  with  only  a  few  changes  to  the  class 
of  Bregman  distances  we  consider. 

Our  approach  differs  from  much  of  the  literature  in  convex  analysis  in  several  ways.  First,  we  work 
primarily  with  the  argument  at  which  a  convex  conjugate  takes  on  its  value,  rather  than  the  value 
of  the  function  itself.  The  reason  for  this  is  that  the  argument  corresponds  to  a  statistical  model, 
which  is  the  main  object  of  interest  in  statistical  or  machine  learning  applications,  while  the  value 
corresponds  to  a  likelihood  or  loss  function.  Second,  while  Bregman  distances  are  typically  defined 
only  on  the  interior  of  the  domain  of  the  underlying  convex  function,  we  assume  that  there  is  a 
continuous  extension  to  the  entire  domain.  This  makes  it  possible  to  formulate  a  very  natural 
duality  theorem  that  also  includes  many  cases  required  in  practice,  when  the  desired  model  may 
lie  on  the  boundary  of  the  domain. 

The  following  section  recalls  the  standard  definitions  from  convex  analysis  that  will  be  required,  and 
presents  the  technical  assumptions  made  on  the  class  of  Bregman  distances  that  we  work  with.  We 
also  introduce  some  new  terminology,  using  the  terms  Legendre-Bregman  conjugate  and  Legendre- 
Bregman  projection  to  extend  the  classical  notion  of  the  Legendre  conjugate  and  transform  to 
Bregman  distances.  Section  3  contains  the  statement  and  proof  of  the  duality  theorem  that  connects 
the  primal  problem  with  its  dual,  showing  that  the  solution  is  characterized  in  geometrical  terms 
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by  a  Pythagorean  equality.  Section  4  defines  the  notion  of  an  auxiliary  function,  which  is  used  to 
construct  iterative  algorithms  for  solving  constrained  optimization  problems.  This  section  shows 
how  convexity  can  be  used  to  derive  an  auxiliary  function  for  Bregman  distances  based  on  separable 
functions.  The  last  section  summarizes  the  main  results  of  the  paper. 


II.  Bregman  Distances  and  Legendre-Bregman  Projections 

In  this  section  we  begin  by  establishing  our  notation  and  recalling  the  relevant  notions  from  convex 
analysis  that  we  require;  the  classic  text  (Rockafellar,  1970)  remains  one  of  the  best  references 
for  this  material.  We  then  define  Bregman  distances  and  their  associated  conjugate  functions  and 
projections,  and  derive  various  relations  between  these  that  will  be  important  in  proving  the  duality 
theorem.  Next  we  state  our  assumptions  on  the  underlying  convex  function  that  enable  us  to  derive 
these  properties  for  the  continous  extension  of  the  Bregman  distance  to  the  entire  domain. 

A.  Notation  and  Basic  Definitions 

We  will  use  notation  that  is  suggestive  of  our  main  applications:  rather  than  4>{x)  we  will  write 
or  (f>(q),  having  in  mind  probability  distributions  p  or  q.  A  convex  function  <f  :  S  C  Rm  — > 
[— oo,+oo]  is  proper  if  there  is  no  gE  S'  with  f(q)  =  — oo  and  there  is  some  q  with  <f{q)  /  oo.  The 
effective  domain  of  denoted  A^,  is  the  set  of  points  where  f>  is  finite:  A^  =  {gE  S  \  cfi(q)  <  oo}; 
for  brevity  we  usually  refer  to  A^  as  simply  the  domain  of  f.  A  proper  convex  function  is  closed  if 
it  is  lower  semi-continuous.  The  conjugate  <f*  of  f  is  given  by 

^(y)  =  sup  ((q,  v)  -  <t>(q))  (2.1) 

q  £  S 

A  proper  convex  function  </>  is  said  to  be  essentially  smooth  or  steep  if  it  is  differentiable  on  the 
interior  of  its  domain  int(A^)  /  0,  and  if  lim,woo  |V^(gn)|  =  +oo  whenever  qn  is  a  sequence  in 
int(A0)  converging  to  a  point  on  the  boundary  of  int(A^).  The  function  f  is  said  to  be  coercive  in 
case  the  level  set  {g  E  S  \  f(q)  <  c}  is  bounded  for  every  cSl. 

Definition  2.1.  Let  f  be  a  closed,  convex  and  proper  function  defined  on  S  C  Mm,  such  that  f 
is  differentiable  on  int(A^)  /  0.  The  Bregman  distance  D ^  :  A^  x  int(A0)  — >  [0,oo)  is  defined  by 

D<f>{p,q)  =  f(p)  -  <t>(q)  -  (V0(g),p- q)  (2.2) 

Bregman  distance  can  be  interpreted  as  a  measure  of  the  convexity  of  f.  This  is  easy  to  visualize 
in  the  one-dinrensional  case:  drawing  a  tangent  line  to  the  graph  of  f>  at  q,  the  Bregman  distance 
D(j)(p,  q)  is  seen  as  the  vertical  distance  between  this  line  and  the  point  <f(p). 

Legendre  functions  are  a  very  well  behaved  family  of  convex  functions  that  will  make  working  with 
Bregman  distances  much  easier. 

Definition  2.2.  A  closed  convex  function  0  is  Legendre,  or  a  convex  function  of  Legendre  type,  in 
case  int(A^)  is  convex  and  f  is  essentially  smooth  and  strictly  convex  on  int(A0). 
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The  primary  properties  that  make  working  with  Legendre  functions  convenient  are  summarized  in 
the  following  results  quoted  from  (Rockafellar,  1970). 

Proposition  2.3.  (Rockafellar,  1970;  Theorem  26.5)  If  (p  is  a  convex  function  of  Legendre  type 
then  V</>  :  int(A^)  — >  int(A^*)  is  a  bijection,  continuous  in  both  directions,  and  V</>*  =  (V0)_1. 

Proposition  2.4.  Suppose  that  <p  is  Legendre,  and  ip  is  a  proper  closed,  convex  function  that  is 
also  essentially  smooth.  Then  (p  +  ip  is  Legendre. 


In  particular,  since  for  fixed  gein^A^,)  the  mapping  p  i — >  (V0(g),p  —  q)  +  j>(q)  is  affine  linear, 
the  function  p  i — >  D^pp,  q)  is  Legendre  with  domain  A^,  and  with  conjugate  domain  A^*  —  Vcp^q). 

Definition  2.5.  For  (f>  a  convex  function  of  Legendre  type  we  define  the  Legendre-Bregman  con¬ 
jugate  :  int(A^)  x  Mm  — >  M  U  {oo}  as 


4(g,u)  =  sup  ((v,p)  -  D^faq))  (2.3) 

a<a 

We  define  the  Legendre-Bregman  projection  :  int(A^)  x  Mm  — >  A^  as 

£<t>(q,v)  =  argrnax  ((v,  p)  —  D(p(p,  q))  (2.4) 


whenever  this  is  well  defined. 


Let  us  explain  our  choice  of  terminology  in  the  above  definition,  which  is  nonstandard.  When  h 
is  a  convex  function  of  Legendre  type,  the  Legendre  conjugate,  as  defined  in  (Rockafellar  1970; 
Chapter  26),  corresponds  to  the  convex  conjugate  h* .  For  fixed  q,  our  definition  of  the  Legendre- 
Bregman  conjugate  is  simply  the  classical  Legendre  conjugate  for  the  convex  function  h(p)  = 
D(f,(p,q )•  Note  that  Rockafellar  defines  the  Legendre  transform  as  the  mapping  from  the  original 
convex  function  (and  domain)  to  its  Legendre  conjugate  (and  associated  domain).  The  Legendre- 
Bregman  projection  C^(q,  v )  is  the  actual  argument  at  which  the  maximum  is  attained.  As  shown 
by  the  following  result,  our  use  of  the  term  “projection”  accords  with  the  standard  terminology  of 
Bregman  projections. 

Proposition  2.6.  Let  (p  be  Legendre.  Then  for  (/Gint(A^)  and  u£int(A^*)  —  X7cp(q),  the 
Legendre-Bregman  projection  is  given  explicitly  by 

C4q,v)  =  (VcP*)(V(P(q)  +  v)  (2.5) 

Moreover,  it  can  be  written  as  a  Bregman  projection 

£<f>(q,v)  =  arg min  L>^(p,g)  (2.6) 

p  6 

for  the  hyperplane  H  =  {p£  Mm  |  (p,  v)  =  6}  with  b  =  (C^q,  v),v). 
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Proof.  To  prove  the  first  statement,  note  that  a  stationary  point  p*  of  (p,  v)  —  D^p,  q)  must 
satisfy: 

V(j){p*)  =  V</>{q)  +  v  (2.7) 

Since  <ft  is  Legendre,  for  V </>((/)  +  v  £  int(A0*),  we  then  have  that 

P*  =  (VtrHVM  +  v)  (2-8) 

=  (V0*)(V#7)+v)  (2.9) 

using  the  fact  that  (V<^>)_1  is  well  defined  and  equal  to  V</>*  from  Proposition  2.3.  The  second 
statement  follows  from,  for  example,  the  results  in  Section  2.2  of  Censor  and  Zenios  (1997)  on 
projections  onto  hyperplanes,  noting  that  every  Legendre  function  is  zone  consistent.  □ 

In  our  formulation  of  the  duality  theorem,  it  is  the  Legendre-Bregman  projection  C<j)(q,v)  rather 
than  the  conjugate  function  £(f,(q1v)  that  plays  a  central  role,  leading  to  a  natural  and  simple 
statement  of  convex  duality  for  Bregman  distances.  This  projection  corresponds  to  a  statistical 
model,  which  is  the  primary  object  of  interest  for  machine  learning  problems. 

B.  Basic  Relations 

We  now  derive  some  basic  algebraic  relations  between  D^,  C<p,  and  These  relations  will  be 
important  in  establishing  the  geometrical  aspects  of  the  duality  theorem,  as  well  as  for  deriving 
auxiliary  functions.  In  order  to  free  us  from  having  to  specify  the  domain  of  and  we  will  in 
the  following  assume  that  A^,*  =  Mm. 

Proposition  2.7.  Let  <f>  be  Legendre,  with  A^*  =  Mm.  For  Gxed  p£  A^,  D^fp,  C^(q,v))  is  con¬ 
tinuous  in  q  Gint(A^)  and  convex  in  v.  Together,  the  Legendre-Bregman  conjugate  and  projection 
satisfy 


D<l>(p,q)  -  D^C^v))  =  {v,p)  -  £^>(q,v)  (2.10) 

=  D(C(t>(q,v),q)  + (v,p- C^v))  (2.11) 

for  all  p  £  A^,  q  £  int(A^)  and  v  £  Mm. 

Proof  Using  the  definition  of  Bregman  distance  and  Proposition  2.6,  we  have  for  <7£int(A^) 

that 


D<j>iP,q)  -  D^ipX^v)) 

=  -#?)  +  (V0(A0(g,?;)),p  -  C^(q,v))  -  (V<p(q),p-  q)  (2.12) 

=  0{£<i>{q,v))  -  </>(<?)  +  (v</>(<?)  +  v,p-  £</,(?, u))  -  (v<£(g),p-  q)  (2-i3) 
=  (v,p)  -  (v,  £</,(?>  v))  +  ^>(C^(q,v))  -  <j>{q)  -  (V<t>(q),£^(q,v)  -  q)  (2.14) 
=  (v,p)  -  (v,  C^q,  v))  +  D^C^v)^)  (2.15) 

=  (v,p)-U(q>v)  (2-16) 
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Therefore  (2.10)  holds  for  pG  A^,  and  g£int(A^).  From  the  definition  of  the  Legendre-Bregman 
conjugate  we  have  for  q  £  int(A^),  that 


^(q,v)  =  (v,£<p(q,v))  -  D^(C^{q,v),q)  (2.17) 

Equation  (2.24)  results  from  combining  this  with  (2.10).  The  convexity  of  D^fp,  £</,((?,  u))  in  v 
follows  from  (2.10),  which  expresses  D^fp,  C^(q,  v))  as  a  sum  of  the  convex  functions  i(p(q,v)  and 
D<j,(p,q)  -  (v,p).  □ 

The  next  result  shows  how  D^fp,  C<f,(q,  v))  varies  with  v,  and  will  be  an  important  ingredient  in 
the  duality  result  of  the  next  section. 

Proposition  2.8.  Let  <f>  be  Legendre,  with  p  £  A^  and  q  £  int(A^).  Then  for  v  £  Mm,  the  mapping 
t  i— >  D^ip,  C^q^tv))  is  differentiable  at  t  =  0,  with  derivative 

Dfiip,  Ctf>(q,  tv))  =  (v,q)  -  (v,p)  .  (2.18) 

t= 0 


d_ 

dt 


Proof.  Since  0  is  Legendre,  if  qGint(A(j>)  then  C^,{q,tv)  £int(A^);  see  for  example  Theorem 
3.12  of  (Bauschke  &  Borwein,  1997).  From  Proposition  2.7  we  have  that 


d_ 

dt 


Dtf)(p ,  C^q,  tv)) 

t= 0 


« tv,/:^(q,tv ))  -  (tv,p)  +  D(j)(C(f,(q,  tv),q)) 


d_ 

dt 


(v,q)  -  (v,p}+ 

(v,q)  -  (v,p) 


</>(£</>(<?, tv))  -  (V(f(q),  4 

fco  dt 


£<t>(q,tv)) 


(2.19) 

(2.20) 
(2.21) 


which  proves  the  result  for  p  £  A^  and  q  £  int(A0).  □ 


C.  The  Continuous  Extension 

The  results  above  are  given  in  terms  of  the  Bregman  distance  using  its  standard  definition  as  a 
function  on  A^  x  int(A^).  We  now  make  assumptions  that  allow  us  to  work  with  D ^  as  an  extended 
real- valued  function  on  A^  x  A^.  This  enables  us  to  formlate  a  very  natural  and  general  duality 
result,  presented  in  the  following  section. 

Informally,  we  assume  that  D ^  extends  continously  from  A^  x  int(A^)  to  A^  x  A^,,  and  that 
extends  continously  from  int(A^)  x  Mm  to  A^  x  Rm.  In  addition,  we  require  a  form  of  compactness 
to  guarantee  the  existence  of  certain  minimizers.  As  before,  in  order  to  simplify  the  presentation 
we  assume  that  the  range  of  V</>  is  all  of  Mm. 
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Thus,  we  make  the  following  assumptions  on  <f\ 


Al.  4>  is  of  Legendre  type; 

A2.  Ar  =  Rm; 

A3.  Dfi  extends  to  a  function  D ^  x  A^  — >  [0,  oo]  such  that  D^p,  q )  is  jointly  continuous  in 
p  and  q,  and  satisfies  D^p,  q)  =  0  if  and  only  if  p  =  q. 

A4.  extends  to  a  function  :  A^  x  Rm  — >  A^  such  that  C^q,  v )  is  jointly  continuous  in  q 
and  v ,  and  satisfies  C^q,  0)  =  q. 

A5.  D^(p,  •)  is  coercive  for  every  p  £  A0\int(A^); 

Note  that  since  for  a  Legendre  function  V0  is  continous  on  int(A^)  (Proposition  2.3),  it  follows 
from  Definition  2.1  that  D^ip.q)  is  jointly  continous  on  A^  x  int(A^)  and  from  Proposition  2.6 
that  £<^(c/,  u)  is  jointly  continous  on  int(A^)  x  Rm.  We  also  note  that  since  we  assume  A^*  =  Rm, 
D(p(p,  •)  is  automatically  coercive  for  pEint(A^).  Together,  conditions  A1-A5  imply  that  </>  is  a 
Bregman-Legendre  function  as  defined  by  Bauschke  and  Borwein  (1997). 

Now,  from  the  definition  of  the  Legendre-Bregman  conjugate  we  have 

^(Q,v)  =  (v,£<f>{q,v))  -  D^C^v)^)  (2.22) 

for  gEin^A^).  Properties  A4  and  A5  allow  us  to  define  :  A^  x  Rm  — >  R  as  the  continuous 
extension  of  :  int(A^)  x  Rm  — >  R,  satisfying  the  same  identity.  Thus,  the  Legendre-Bregman 
conjugate  v )  is  continuous  in  q,  continuous  and  convex  in  v ,  and  satisfies  i^q,  0)  =  0. 

Proposition  2.7  now  generalizes  to  the  continuous  extension. 

Proposition  2.9.  Let  f>  satisfy  A1-A4.  For  fixed  p  £  A^,  Drf,(p,  C^q,  u))  is  continuous  in  q  and 
convex  in  v.  Together,  the  Legendre-Bregman  conjugate  and  projection  satisfy 

D<p(p,  q)  —  D^p,  C^q,  v))  =  (v,p)  -  i^q,  v)  (2.23) 

=  D{£(j){q,v),q)  +  (v,p-  C<j,(q,v))  (2.24) 

for  all  p,q£  A $  and  v  £  Rm. 


Proof.  This  follows  directly  from  the  joint  continuity  of  D and  □ 

The  differential  identity  in  Proposition  2.8  also  extends. 


Proposition  2.10.  Let  < j>  satisfy  A1-A4,  and  let  p,q£  A^  with  D^fp,  q)  <  oo.  Then  for  v  £  Rm, 
the  mapping  t  e- >  D^p,  C<j,(q,tv))  is  differentiable  at  t  =  0,  with  derivative 

D(j>(p ,  T,f,(q,tv))  =  (v,q)  -  (v,p)  .  (2.25) 

t= 0 


d_ 

dt 
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(2.26) 


Proof.  Let  gGint(A^).  From  Proposition  2.8  we  know  that 

D<t>(p,C<t,(q,tv))  =  (v,q)-(v,p) 

t= 0 

Thus,  since  C^q,  ( t  +  s)v )  =  C^C^q,  tv),  sv ),  we  also  have  that 

^D(p,C(j,(q,tv))  =  (v,C^(q,tv))  -  (v,p)  (2.27) 

To  show  that  the  result  holds  when  q  G  A^\int(A0),  we’ll  use  a  fact  from  elementary  analysis:  if 
fn  —>  f,  and  f'n  is  continuous  with  f'n(t)  —>  g(t)  uniformly  for  f  G  [a,  6],  then  g  is  continous  and 
f'(t)  =  g(t).  First,  let  gGint(A^)  and  p  ^  int(A^).  Suppose  pnGint(A0)  with  pn  — >  p.  Since 
£^,(g,tu)  Gint(A0),  we  have  from  the  above  calculation  that 

^j_D{pn,C(j){q,tv))  =  (viC^tv))  -  {v,pn)  — *  (viC^tv))  -  (v,p)  (2.28) 

where  the  convergence  is  uniform  on  every  interval  [a,b\  around  zero;  property  (2.25)  follows. 

Now  suppose  that  pG  A^,  q  G  A0\int(A0),  and  gnGint(A0)  with  qn  —>  q.  Because  C^q,  v)  is 
jointly  continous  in  q  and  v,  it  is  uniformly  continuous  on  every  compact  set  of  ( q ,  v).  In  particular, 
(v,  C^qn,  tv))  —  (v,  p)  converges  uniformly  in  t  to  (v,  C^q,  tv))  —  (v,  p)  on  every  interval  [a,  b] .  Thus 
property  (2.25)  holds  for  all  p,  q  G  A^.  □ 

Proposition  2.9  and  2.10  are  the  main  computations  that  we  will  require  in  the  following  section. 


d_ 

dt 


III.  Duality 

In  this  section  we  derive  the  main  duality  result.  The  setup  is  that  we  have  a  set  of  features 
/(i)Glm,  j  =  1,2,...  , n  and  denote  by  F  the  mxn  matrix  with  columns  given  by  the 
These  features  correspond  to  the  “weak  learners”  in  boosting,  or  to  the  sufficient  statistics  in  an 
exponential  model.  The  primal  problem  constrains  the  values  Op,  f  ^),  and  these  constraints  carry 
over  to  Lagrange  multipliers  in  a  family  of  Legendre-Bregman  projections  C(q,F\)  in  the  dual 
problem. 

Definition  3.1.  For  a  given  element  p o  G  A^,  the  feasible  set  for  po  and  F  is  defined  by 

V(p0,F)  =  {pGA^  |  (p,/^)  =  (po,/(j)),  j  =  1,...  n}  (3.1) 

For  a  given  go  G  A^,  the  Legendre-Bregman  projection  family  for  go  and  F  is  defined  by 

Q{qo,F)  =  {<?e  A^  I  g  =  C<p{qo,  FX)  for  some  A  G  Mn}  (3.2) 

Trivially,  both  sets  are  non-empty  since  po  G  V(po,  F)  and  go  G  Q(qo,F).  Since  po,  go,  and  F  will 
be  fixed,  we  will  use  abbreviated  notation  and  refer  to  these  sets  as  V  and  Q.  We  use  Q  to  denote 
the  closure  of  Q(go,  F)  as  a  subset  of  Mm.  Duality  relates  the  projection  onto  V  to  the  projection 
onto  Q. 
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Proposition  3.2.  Let  f>  satisfy  A1-A5,  and  suppose  that  po,  qo  G  with  D(p(po,  go)  <  oo.  Then 
there  exists  a  unique  g*  G  satisfying  the  following  four  properties: 

(1)  g*G?nQ 

(2)  D^ip,  q)  =  D^p,  g*)  +  D^q*,  q)  for  any  peV  and  q<EQ 

(3)  q *  =  argmin  D^p,  qQ) 

jjGP 

(4)  q *  =  arg  min  D^po,  q) 

q  GQ 

Moreover,  any  one  of  these  four  properties  determines  q *  uniquely. 

In  order  to  prove  Proposition  3.2,  we  first  prove  two  lemmas.  The  first  shows  that  there  is  at  least 
one  member  in  common  between  V  and  Q;  the  second  shows  that  the  Pythagorean  equality  (2) 
holds  for  any  such  member. 

Lemma  3.3.  If  D^Jpo,  q0)  <  00  then  V  fi  Q  is  nonempty. 

Proof.  Note  that  since  D^po,  qo)  <  00,  D^po,  q)  is  not  identically  00  on  Q.  Also,  the  mapping 
A  D(j)(po,  T<f,(qo1  FX))  is  continuous  and  convex.  Let  IZ  be  the  level  set 

11  =  {gGA^  I  £>4p0,g)  <D(po,qo)}  (3.3) 

We  know  from  Assumption  A5  that  IZ  is  bounded.  Thus  D^(po,q)  attains  its  minimum  at  a  (not 
necessarily  unique)  point  g*  G  Q  fl  IZ  C  Q.  We  will  show  that  g*  is  also  in  V. 

Let  qpQ,  and  let  pj  G  Rn  be  such  that  g  =  linr^oo  /^(go,  Fpj).  Then  by  the  continuity  of  £</,(-,  ‘)> 

FX)  =  lim  £0(£^(go,F^),FA)  (3.4) 

j->° o 

=  lim  T</,(go,  F(pj  +  A))  G  Q  (3.5) 

j^oo 

Thus  Q  is  closed  under  the  mapping  q  1— >  C^(q,FX)  for  AgR™,  and  £0(g*,F A)  is  in  Q  for  any  A. 
By  the  definition  of  g*,  it  follows  that  A  =  0  is  a  minimum  of  the  function  A  D(j)(po,  C^q*,  FX)). 
Taking  derivatives  with  respect  to  A  and  using  Proposition  2.10  we  conclude  that  (g*,  /)  =  ( po ,  /}; 
thus  g*  G  V.  □ 

Lemma  3.4.  If  g*  G  V  (~l  Q  then  the  Pythagorean  equality  D^p ,  g)  =  D^p ,  g*)  +  D^(g*,  g)  holds 
for  any  p  G  V  and  q  €  Q. 


Proof.  Suppose  that  p\,p2,  <?i,  q 2  G  A^,  with  g2  =  C(p(qi,  FX).  From  Proposition  2.9  we  have 

that 

D^ipi,  qi)  -  D^ipi,  q2)  =  (pi,  FX)  -  £<j,(pi,  FX)  (3.6) 


and  similarly 


D<p(P2,qi)  ~  D<j,(p2,q2)  =  {P2,FX}  -e<j)(p2,F\) 


(3.7) 


Therefore, 

D^>(pi,qi)  -  D<j>(p1,q2)  -  D(j){p2,qi)  +  D^{p2,q2)  =  (pi,FX)  -  (p2,  FX)  (3.8) 

n 

=  Xi  ((P1’  /W)  ~  (P2j  /W)) 
i=i 

It  follows  from  this  identity  and  the  continuity  of  D $  that 

F><f,{p  1,  qi)  -  D^ijp i,  g2)  -  A/>(P2,  ?i)  +  A/>(p2,  ?2)  =  0  (3.9) 

if  PhP2  G  F  and  gi,  g2  £  Q-  The  lemma  follows  by  taking  pi  =  gi  =  g*.  □ 

Proof  of  Proposition  3.2.  Choose  g*  to  be  any  point  in  VnQ.  Such  a  g*  exists  by  Lemma  3.3. 
It  satisfies  property  (1)  by  definition,  and  it  satisfies  property  (2)  by  Lemma  3.4.  As  a  consequence 
of  property  (2),  it  also  satisfies  properties  (3)  and  (4).  To  check  property  (3),  note  that  if  g  is  any 
point  in  Q.  then  D^po,  q)  =  D^po,  g*)  +  -D^(g*,  g)  >  D(j,(p{hq+).  Similarly,  property  (4)  must  hold 
since  if  p  is  any  point  in  V.  then  D^{p,  g0)  =  D^p,  g*)  +  L>^(g*,  g0)  >  L>(g*,  go)  • 

It  remains  to  prove  that  each  of  the  four  properties  (1)— (4)  determines  g*  uniquely.  In  other  words, 
we  need  to  show  that  if  m  is  a  point  in  satisfying  any  of  the  four  properties  (1)— (4),  then 
m  =  g*.  Suppose  that  m  satisfies  property  (1).  Then  property  (2)  with  p  =  q  =  m  implies  that 
D^fm^m)  =  D^m,  g*)  +  D^,(g*,m).  Since  D^m,  rn)  =  0,  it  follows  that  g*)  =  0  so  m  =  g*. 

If  m  satisfies  property  (2),  then  the  same  argument  with  g*  and  m  reversed  proves  that  m  =  g*. 
Suppose  that  m  satisfies  property  (3).  Then 

D^ipo,  g*)  >  D<f,(p0,  m)  =  D^po,  g*)  +  L>0(g*,  m)  (3.10) 

where  the  second  equality  follows  from  property  (2)  for  g*.  Thus  Drf)(qir,  rn)  <  0  so  m,  =  g*.  If  rn 
satisfies  property  (4),  then 

A/>(g*,  go)  >  g0)  =  D^m,  g*)  +  A/,(g*,  go)  (3.11) 

showing  once  again  that  m  =  g*.  □ 

In  the  following  section  we  outline  the  auxilary  function  method  for  building  iterative  algorithms  to 
compute  g*.,  and  show  how  to  use  convexity  to  derive  an  auxiliary  function  for  separable  Bregman 
distances. 

IV.  Auxiliary  Functions 

The  auxiliary  function  approach  is  conceptually  simple:  bound  the  change  in  Bregman  distance 
from  below  using  a  function  that  is  easy  to  compute  and  that  decouples  the  constraints.  Maximizing 
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this  auxiliary  function  we  obtain  new  parameters  X'  =  A  +  AA  and  a  new  model  qx+AX  given  by 


q\+AX  =  C'piqx,  F AX) 

(4.1) 

=  C4>{C(j>{qo,FX),FAX) 

(4.2) 

=  £</>(<7o>  F(X  +  AA)) 

(4.3) 

We  then  use  the  duality  theorem  to  show  that  when  AA  =  0,  we  must  have  that  q  =  q *. 

The  strategy  is  very  similar  to  EM.  In  an  EM  algorithm,  the  Q-function  Q(A',  A)  is  computed  as  a 
lower  bound  to  the  change  in  log- likelihood: 

,  s,  g(adA')  v-^  ,  ..  ^2hl(x,h\X') 

Po{x)log———  =  \  p0{x)log^ — — — - -  (4.4) 

q{x  |  A)  ^  q(x  |  A) 

=  log  I  A)  9q(x!h\X)  (45) 

>  ^q(h\XlX)log^^  (4.6) 

=  Q(A',A)  (4.7) 

where  (x,  h )  is  the  complete  data,  x  is  the  incomplete  data,  and  the  inequality  follows  from  the 

concavity  of  the  logarithm.  After  computing  Q  in  the  E-step,  it  is  then  maximized  over  X'  in  the 
M-step. 

In  the  same  way,  for  Bregman  distances  the  aim  is  to  derive  an  auxiliary  function  A(A/,  A)  whose 
calculation  can  be  carried  out  efficiently  in  something  like  an  E-step,  and  such  that  it  can  be  easily 
maximized  over  X'  in  an  M-step.  However,  just  as  for  EM,  this  is  a  general  strategy  more  than  it  is 
a  precise  algorithm.  A  particular  Bregman  distance  problem  may  require  some  ingenuity  in  order 
to  come  up  with  an  appropriate  auxiliary  function. 

This  is  the  general  motivation  behind  the  following  two  definitions. 

Definition  4.1.  A  function  A  :  A^  x  Mn  — >  M  is  called  an  auxiliary  function  for  po  and  F  in  case 

1.  A(q,  A)  is  continuous  in  q  and  A(q,  0)  =  0 

2.  D^po,  q)  —  D^po,  C^q,  FX))  >  A(q,  A) 

3.  If  A  =  0  is  a  maximum  of  A(q ,  A),  then  ( q ,  /Cfi)  =  /^)  for  j  =  1, . . . ,  n. 

Definition  4.2.  Let  A  he  an  auxiliary  function  and  q oGA^.  The  update  sequence  for  q$  with 
respect  to  A  is  defined  by  =  qo  and 

q{t+ 1)  =  £0(gW,FA(t))  where  X®  =  arg  maxA  A{ q ^ ,  A)  (4-8) 

The  reason  for  defining  auxiliary  functions  in  this  way  is  the  following  result,  which  can  be  proved 
in  a  similar  way  to  Proposition  5  in  (Della  Pietra  et  al.,  1997)  or  Lemma  2  in  (Collins  et  al.,  2001). 
The  compactness  assumption  will  in  general  follow  from  coercivity  in  Assumption  A5. 
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Proposition  4.3.  Suppose  that  the  sequence  q1'1'1  lies  in  a  compact  set.  Then 

lim  =  argmin D(f,(po,  q)  (4.9) 

t~°°  q  GQ 

As  we  now  explain,  auxiliary  functions  can  be  conveniently  constructed  by  using  the  relation 

D(j){p,  q)  -  £</,(?,  v))  =  (P,v)  -l<j>{q,v)  (4.10) 

from  Proposition  2.9  and  exploiting  the  convexity  of  i^{q,v).  For  simplicity,  we  will  assume  that 
cj)  is  separable,  so  that  fap)  =  ^2ifa{Pi)  with  each  fa  :  M  — >  M  satisfying  properties  A1-A5  (with 
m  =  1).  Auxiliary  functions  in  the  general  case  can  be  derived  using  similar  arguments.  For  the 
separable  case,  clearly 


m 


D<f,(p,  q) 

1=1 

(4.11) 

S 

c? 

«WJ 

li 

(4.12) 

where  ^(q,  v)  =  supp  ^  (pv  —  D^fp,  q ))  is  the  Legendre-Bregman  conjugate  of  <fo. 


Proposition  4.4.  For  each  i  =  1  select  IVj  so  that  (Cj=i  |//^|  <  Aj,  and  set  Sij  = 

sign(/(^).  Then 

IT  7YL  TL 

A(q,  A)  d=f  A,  (po,  fW)  -  -L  J2  SijNiXj)  (4.13) 

j= 1  i=  1  1  3= 1 

is  an  auxiliary  function  for  po  and  F,  and  the  corresponding  update  scheme  is  given  by 

qV+V  =£f(q(t),FX®)  (4.14) 

where  A^  satisfies 

m 

fij)  C-bM('i\sijNi\j)  =  (po,  /0))  (4.15) 


The  idea  of  factoring  out  the  signs  is  taken  from  Collins  et  al.  (2001).  If  we  take  IVj  =  1  for 
all  i  we  obtain  the  algorithms  presented  in  that  paper.  If  we  take  IVj  =  y~b  \f-^\  then  we  obtain 
algorithms  analogous  to  the  IIS  algorithm  of  Della  Pietra  et  al.  (1997). 

Proof.  We  verify  that  the  function  defined  in  (4.13)  satifies  the  three  properties  of  Defini¬ 
tion  4.1.  Property  (1)  of  the  definition  holds  since  i(pi  ((/,;,  0)  =  0.  Property  (2)  follows  from  the 
convexity  of  In  particular,  we  have  the  inequality 

m 

U(q,FX)  =  )i)  (4.16) 

i=  1 


li 


(4.17) 


_  ’  yi  Sii  i  i  ) 

i=l  1=1 


*=i  \l=i 


l=i 


JVi 


IF, 


yy  yy  f 7  £<t>i{<iiisijNi\j) 


.  i  .  ,  iV, 

*=i  j=i 

which  together  with  Proposition  2.9  says  that 


(4.18) 

(4.19) 


D<t,{p,q)  -D^{p,C^{q,F\))  >  ^  Aj  ^p,  /(j))  -  £&(?*,  s^Xj)  (4.20) 


1=1 


*=1  1  =  1 


Now,  using  Propositions  2.9  and  2.10,  it  can  be  shown  that 

J^i(tt,v)  =  £<t>Mi-,v)  (4.21) 

which  shows  that  (4.15)  is  the  correct  update.  Therefore,  at  a  maximum  A*  of  A(q,  A)  we  have  that 

m 

^  SijNiX*)  for  each  j  (4.22) 

i=  1 

If  A*  =  0,  then 

m 

(po,f(j))  =  0)  =  (<Z,/(i))  (4.23) 

1=1 

showing  that  Property  (3)  holds.  Thus  (4.13)  defines  an  auxiliary  function.  □ 

While  the  auxiliary  function  (4.13)  looks  a  bit  messy,  both  the  “E-step”  and  “M-step”  for  this  type 
of  auxiliary  function  are  generally  quite  practical  and  easy  to  implement. 


V.  Conclusion 

This  paper  has  presented  a  convex  duality  theorem  for  constrained  optimization  using  Bregman 
distances.  The  main  result,  Proposition  3.2,  differs  from  results  presented  in  the  convex  analysis 
literature  in  that  the  Bregman  distance  is  defined  on  the  entire  essential  domain,  rather  than  only 
on  the  interior.  This  generality  is  needed  in  many  applications.  Though  the  assumptions  A1-A5 
that  we  make  on  the  underlying  convex  function  are  fairly  restrictive,  it  may  well  be  possible  to 
relax  these  assumptions  to  cover  a  broader  class  of  examples.  In  particular,  assumption  A2,  which 
states  that  the  conjugate  domain  is  all  of  Mm,  may  not  be  essential  in  our  approach. 

The  auxiliary  function  technique  presented  in  Section  4  is  a  general  and  practical  method  for 
deriving  algorithms  for  solving  the  dual  problem.  Although  the  specific  auxiliary  function  we 
derive  assumes  the  Bregman  distance  is  separable,  similar  arguments  can  be  used  for  non-separable 
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Bregman  distances.  The  analysis  given  here  makes  clear  the  role  of  convexity,  as  the  bounds  are 
derived  using  only  the  properties  of  the  underlying  Legendre-Bregman  conjugate. 
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